Experiences and issues using the HandPose model in ml5.js
Following questions lead into testing ml5.js:
Can you build an Interface that gets controlled by the users webcam input? Does ml5.js and their hand-tracking model work for that?
Test: Building a small UI controlled by users finger tips 👆
- Webcam input gets loaded into p5.js
- Webcam gets send to ml5.js
- ml5.js returns Object with position data of points on your hands
- take only the finger tip x and y position
- check if those positions are in between the ui fields
- if so change their color (or do anything else 🙂)
Issue 1: Camera takes seconds to load
This is acceptable for Installations that get started once, but if you implement this in a web page users will need to wait. I am not sure why that is. I have tried to constrain the video input in its dimensions and framerate but that didn't change much.
let constraints = {
video: {
width: { min: 640, ideal: 640 },
height: { min: 480, ideal: 480 },
frameRate: { max: 25 },
}
};
Is it p5.js or is it a problem with some WebAPI?
Issue 2: Performance issues with the tfjs model
There were strong performance issues with the tfjs model. First of all it took a long time to load and it was also stuttering. It was tested on a good Laptop and an even better PC with the same results. This is strange, because in Video tutorials it seams to work for people. Using the models live demo also works fine. But why doesn't it in this test with p5.js and ml5.js?
More info on options here for "tfjs" runtime.
Finding Solutions
What helped was changing the runtime model to the older mediapipe and switching from full to light mode. Light mode runs a smaller version of the model which is enough for just tracking a finger tip. And also the mediapipe model returned much more satisfactory results. It loads much quicker and has no stuttering. Sometimes it does detect other parts of the hand as a finger tip and glitches. This is most probably because it is an older and less trained model. In the end mediapipe did the job with acceptable performance and just some glitching.
let options = {
maxHands: 2,
flipped: true,
runtime: "mediapipe",
modelType: "lite",
};
More info on options here for "mediapipe" runtime.
2026-05-31
My first impression is that it technically works, but the AI tracking seams to be really slow. With such delays it's kind of not usable. I will research on methods to speed up the ml5.js processes. Maybe we can run it locally?
Experiences and issues using the HandPose model in ml5.js
Following questions lead into testing ml5.js:
Can you build an Interface that gets controlled by the users webcam input? Does ml5.js and their hand-tracking model work for that?
Test: Building a small UI controlled by users finger tips 👆
- Webcam input gets loaded into p5.js
- Webcam gets send to ml5.js
- ml5.js returns Object with position data of points on your hands
- take only the finger tip x and y position
- check if those positions are in between the ui fields
- if so change their color (or do anything else 🙂)
Issue 1: Camera takes seconds to load
This is acceptable for Installations that get started once, but if you implement this in a web page users will need to wait. I am not sure why that is. I have tried to constrain the video input in its dimensions and framerate but that didn't change much.
let constraints = {
video: {
width: { min: 640, ideal: 640 },
height: { min: 480, ideal: 480 },
frameRate: { max: 25 },
}
};
Is it p5.js or is it a problem with some WebAPI?
Issue 2: Performance issues with the tfjs model
There were strong performance issues with the tfjs model. First of all it took a long time to load and it was also stuttering. It was tested on a good Laptop and an even better PC with the same results. This is strange, because in Video tutorials it seams to work for people. Using the models live demo also works fine. But why doesn't it in this test with p5.js and ml5.js?
More info on options here for "tfjs" runtime.
Finding Solutions
What helped was changing the runtime model to the older mediapipe and switching from full to light mode. Light mode runs a smaller version of the model which is enough for just tracking a finger tip. And also the mediapipe model returned much more satisfactory results. It loads much quicker and has no stuttering. Sometimes it does detect other parts of the hand as a finger tip and glitches. This is most probably because it is an older and less trained model. In the end mediapipe did the job with acceptable performance and just some glitching.
let options = {
maxHands: 2,
flipped: true,
runtime: "mediapipe",
modelType: "lite",
};
More info on options here for "mediapipe" runtime.
2026-05-31
My first impression is that it technically works, but the AI tracking seams to be really slow. With such delays it's kind of not usable. I will research on methods to speed up the ml5.js processes. Maybe we can run it locally?